Weighted and Probabilistic Context-Free Grammars Are Equally Expressive

This paper studies the relationship between weighted context-free grammars (WCFGs), where each production is associated with a positive real-valued weight, and probabilistic contextfree grammars (PCFGs), where the weights of the productions associated with a nonterminal are constrained to sum to one. Since the class of WCFGs properly includes the PCFGs, one might expect that WCFGs can describe distributions that PCFGs cannot. However, Chi (1999) and Abney, McAllester, and Pereira (1999) proved that every WCFG distribution is equivalent to some PCFG distribution. We extend their results to conditional distributions, and show that every WCFG conditional distribution of parses given strings is also the conditional distribution defined by some PCFG, even when the WCFG’s partition function diverges. This shows that any parsing or labeling accuracy improvement from conditional estimation of WCFGs or CRFs over joint estimation of PCFGs or HMMs is due to the estimation procedure rather than the change in model class, since PCFGs and HMMs are exactly as expressive as WCFGs and chain-structured CRFs respectively.